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We present exact results for two complementary measures 
of spatial structure generated by ID spin systems with finite- 
range interactions. The first, excess entropy, measures the 
apparent spatial memory stored in configurations. The sec- 
ond, statistical complexity, measures the amount of memory 
needed to optimally predict the chain of spin values. These 
statistics capture distinct properties and are different from 
existing thermodynamic quantities. 
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Thermodynamic entropy, a measure of disorder, is a fa- 
miliar quantity that is well-understood in almost all sta- 
tistical mechanical contexts. It's notable, though, that 
complementary and similarly general measures of struc- 
ture and pattern are largely missing from current theory 
and are certainly less well-developed. To date, "struc- 
ture" has been handled on a case by case basis. Or- 
der parameters and structure functions, for example, are 
typically invented to capture the significant features in 
a specific phenomenon. There is no generally accepted 
approach to answering relatively simple questions, such 
as. How much temporal memory is used by a process to 
produce a given level of disorder? 

In the following we adapt two measures of structure, 
the excess entropy E and the statistical complexity C^, 
to analyze the spatial configurations generated by sim- 
ple spin systems. These measures of structure are not 
problem-specific — they may be applied to any statistical 
mechanical system. We give exact results for E and C^ 
as a function of temperature, external field, and coupling 
strength for one-dimensional finite-range systems. Our 
results show that E and C^ are different from measures 
of disorder, such as thermodynamic entropy and temper- 
ature; rather E and C^ quantify significant aspects of 
information storage and computation embedded in spa- 
tial configurations. 

In our analysis we introduce purely information theo- 
retic coordinates — a plot of E and C^ vs. spatial entropy 
density /i^ - known as the complexity-entropy diagram. 
The benefit of this view is that it is explicitly independent 
of system parameters and so allows very different systems 
to be compared directly in terms of their intrinsic infor- 
mation processing. In past work the complexity-entropy 
diagram was analyzed for a class of processes in which 
the set of allowed configurations changed as a function of 



a system control parameter |l|. For the systems consid- 
ered here, the variation in E and C^ is driven instead by 
the "thermalization" of the configuration distribution. 

Consider a one-dimensional chain of spin variables 
s = . . . s-2 S-i So si ... where Si range over a finite set 
A. Divide the chain into two semi-infinite halves by 
choosing a site i as the dividing point. Denote the left 
half by Si = ... Si_3 Si_2 Si-i Sj and the right half by 
s i= Si_|_i Si+2 Si+3 ... . Let Pr(si) denote the proba- 
bility that the i— variable takes on the particular value 
Si and Pr (si, Si+i, . . . , Si+i) the joint probability over 
blocks of L consecutive spins. Assuming spatial transla- 
tion symmetry, Pr (si, . . . , s^+l) = Pr (si, . . . , sl). 

Given such a distribution one measures the average 
uncertainty of observing a given L-spin block s^ by the 
Shannon entropy H 



Pr(si,...,si,)log2Pr(si,. 
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The spatial density of Shannon entropy of the spin con- 
figurations is defined by h^ = \\ni]^^^L~^H{L). h^ 
measures the irreducible randomness in the spatial con- 
figurations. For physical systems it is, up to a multi- 
plicative constant, equivalent to thermodynamic entropy 
density. It is also equivalent to the average of the config- 
urations' Kolmogorov-Chaitin complexity. As such, /i^ 
measures the average length (per spin) of the minimal 
universal Turing machine program required to produce a 
typical configuration [^,|j . 

The entropy density is a property of the system as 
a whole; only in special cases will the isolated-spin un- 
certainty i?(l) be equal to h^. It is natural to ask, 
therefore, how random the chain of spins appears when 
finite-length spin blocks are considered. This is given 
by hfj.{L) = H{L) — H{L — 1), the incremental increase 
in uncertainty in going from (L — l)-blocks to L-blocks. 
h^{L) overestimates the entropy density h^ by an amount 
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' hfj^ that indicates how much more random the fi- 



nite L blocks appear than the infinite configuration s . 
In other words, this excess randomness tells us how much 
additional information must be gained about the configu- 
rations in order to reveal the actual per-spin uncertainty 



h^. Summing up the overestimates one obtains the total 
excess entropy 
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Informally, E is the amount (in bits), above and be- 
yond /i^, of apparent randomness that is eventually "ex- 
plained" by considering increasingly longer spin-blocks. 
This follows from noting that E may be expressed as the 
mutual information / yj between the two semi-infinite 

halves of a configuration; E = I{ s ; s). That is, E mea- 
sures how much information one half of the spin chain 
carries about the other. In this restricted sense E mea- 
sures the spin system's apparent spatial memory. If the 
configurations are perfectly random or periodic with pe- 
riod 1, then E vanishes. Excess entropy is nonzero be- 
tween the two extremes of ideal randomness and trivial 
predictability. 

Another, related, approach to spatial structure begins 
by asking a different question. How much memory is 
needed to optimally predict configurations? Restated, we 
are asking to model the system in such a way that the 
observed configurations can be statistically reproduced. 
To address this, we must determine the effective states 
of the process; how much of the left configuration must 
be remembered to optimally predict the right? The an- 
swer to these questions leads us to define the statistical 
complexity C^ . 

Consider the probability distribution of all possible 
right halves s conditioned on a particular left half, Si at 
site i: Pr( s \ Si). These conditional probabilities allow 
one to optimally predict configurations. We now use this 
form of conditional probabilities to define an equivalence 
relation ^ on the space of all left halves; the induced 
equivalence classes are subsets of the space of all allowed 
Si . We say that two configurations at different lattice 
sites are equivalent if and only if they give rise to an 
identical conditional distribution of right-half configura- 
tions. Formally, we define the relation ^ by 



'Si iff Prf s 
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The equivalence classes induced by this relation are called 
causal states and denoted Si. Two s belong to same 
causal state if, as measured by the probability distri- 
bution of subsequent spins conditioned on having seen 
that particular left-half configuration, they give rise to 
exactly the same degree of certainty about the configu- 
rations that follow to the right. 

Once the set {Si} of causal states has been identified, 
we can inductively obtain the probability Pr(<Si) of find- 
ing the chain in the i— causal state by observing many 
configurations. Similarly, we can obtain transition prob- 
abilities T between states. The set {Si} together with 
the dynamic T constitute a model — referred to as an e- 
machine |l| — of the original infinite configurations. 



To predict, as one scans from left to right, the succes- 
sive spins in a configuration with an e-machine, one must 
track in which causal state the process is. Thus, the infor- 
mational size of the distribution over causal states gives 
the minimum amount of memory needed to optimally 
predict the right-half configurations. This quantity is 
the statistical complexity 



C^ = -^Pr(5,)log2Pr(5,) 

{Si} 



(4) 



The excess entropy sets a lower bound on the statisti- 
cal complexity: E < C^ [Q. That is, the memory needed 
to perforin optimal prediction of the right-half config- 
urations cannot be lower than the mutual information 
between left and right halves themselves. This relation- 
ship reflects the fact that the set of causal states is not 
in one-to-one correspondence with L-block or even oo- 
length configurations. In the most general setting, the 
causal states are a reconstruction of the hidden, effective 
states of the process. 

Note that for both C^ and E no memory is expended 
trying to account for the randomness or, in this case, 
for thermal fiuctuations present in the system. Thus, 
these measures of structural complexity depart markedly 
from Kolmogorov-Chaitin complexity which demands a 
deterministic accounting for the value of every spin in a 
configuration. As noted above, the per-spin Kolmogorov- 
Chaitin complexity is /i^ PiHl- Finally, note that C^ 
and E follow directly from the configuration distribu- 
tion; their calculation doesn't require knowledge of the 
Hamiltonian. 

As is well known, the partition function for any one- 
dimensional, discrete spin system with finite range inter- 
actions can be expressed in terms of the transfer matrix 
V [^. Using V, we have calculated exact expressions 
for Cp and E for such systems. In the following let v^ 
{u'~) denote the normalized right (left) eigenvector cor- 
responding to F's largest eigenvalue A. 

The first step is to determine the causal states. Con- 
sider an Ising system with nearest neighbor (nn) interac- 
tions. The nn interactions and the fact that a configura- 
tion's probability is determined by the temperature and 
its energy means that only the rightmost spin in the left 
half infiuences the probability distribution of the spins 
in the right half. Thus, the possible causal states are in 
a one-to-one correspondence with the different values of 
a single spin. (This indicates how this class of spin sys- 
tems is a severely restricted subset of e-machines.) This 
observation determines an upper bound for a spin 1/2 nn 
system: C^ < log2 2 = 1. 

To complete the determination of the causal states 
we must verify that conditioning on different spin val- 
ues leads to different distributions for s ; otherwise they 
fall into the same equivalence class and there would be 
only one causal state. This distinction is given by eq. (0) 
which, in terms of the transfer matrix V , reads 



{ufr^v^u + iuf)-%k v*^j 



(5) 



If cq. (||) is satisfied, then 



C^ = -W>fe log2(WfeO • 



(6) 



(In eq. (|6|) and the following, a summation over re- 
peated indices is implied.) For a nn system, eq. (o) is 
equivalent to saying that C^ = -ff(l), the entropy as- 
sociated with the value of one spin. By determining 
an expression for H(L), one sees that /i^ is given by 
hf^ = log A - X^^uf-u^Vki log[ Vfci ] , and that E is given 
by 



-B = -logA + Ju'^uj;Vk^ log[yfe 
wfu^log[w^uf ] , 



(7) 



Note that these results prove an explicit version of the 
inequality between E and C^j mentioned above; namely. 



Ca 



E + h„ 



(8) 



again assuming that eq. (^) is satisfied J^ . 

Let us illustrate the content of eq. (|^by considering 
a special case, a spin 1/2 paramagnet (PM), where there 
are no couplings between spins. Since there are no corre- 
lations between spins, E vanishes. The probability dis- 
tribution of the right-half configuration is independent 
of the left-half configuration. Thus, there is a single, 
unique distribution Pr( s | s ) and eq. (0) is not satis- 
fied. The PM has only one causal state and so C^ = for 
all temperatures. This example shows how the process of 
determining causal states ensures statistical complexity 
measures structure and not randomness. 
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FIG. 1. Cfi, E, and h^ as a function of T for the nn spin 
1/2 ferromagnet. B was held at 0.30 and J — 1. 

Now consider the spin 1/2, nearest neighbor Ising sys- 
tem with Hamiltonian 



H 



-j^s.,sj+i - -sy^. 



(9) 



where, as usual, J is a parameter determining the 
strength of coupling between spins, B represents an ex- 
ternal field, and Si € {-1-1, —1}. 

For all temperatures except zero and infinity eq. dq) is 
satisfied and the causal states are in a one-to-one corre- 
spondence with the values of a single spin. At T = cx3 the 



system is identical to a paramagnet and C^ and E both 
vanish. At T = the system is frozen in its spatially pe- 
riodic ground state; E — C^ — logj P = 0, where P(= 1) 
is the period of the spatial pattern. 

Using eqs. (^) and (||), fig. 1 plots C^ and E a.s & 
function of temperature T. The coupling is ferromag- 
netic (J = 1) and there is a non-zero external field 
{B — 0.3). As expected, the entropy density is a mono- 
tonic increasing function of temperature. Somewhat less 
expectedly (cf . ref. 0| ) , the statistical complexity also in- 
creases monotonically (until T — oo). The excess entropy 
E vanishes gradually in the high and low temperature 
limits. 
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FIG. 2. The complexity-entropy diagram for a ferromagnet 
(FM), an anti- ferromagnet (AFM) and a paramagnet (PM): 
Cf_i and E plotted parametrically against /i^. For a given J, B 
was held constant— B = 0.30 (FM) and B = 1.8 (AFM)— as 



T was varied. All systems have C^ 
denoted by the square token. 
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Figure 2 presents the complexity-entropy diagram for 
a ferromagnet (FM), an anti- ferromagnet (AFM), and 
a paramagnet (PM): C^ and E plotted parametrically 
as a function of ft,^. The diagram gives direct access 
to the information processing properties of the systems 
independent of control parameters (i.e. B, J, and T). 

For the ferromagnet, E is seen to have a maximum in a 
region between total randomness (/i^ = 1) and complete 
order (h^ = 0)- At low temperatures (and, hence, low 
hf^) most of the spins line up with the magnetic field. At 
high temperatures, thermal noise dominates and the con- 
figurations are quite random. In both regimes one half of 
a configuration contains very little information about the 
other half. For low h^ , the spins are fixed and so there is 
no information to share; for high /i^, there is much infor- 
mation at each site, but it is uncorrelated with all other 
sites. Thus, the excess entropy is small in these temper- 
ature regimes. In between the extremes, however, E has 
a unique maximum at the temperature where spin cou- 
pling strength balances the thermalization. The result is 
a maximum in the system's spatial memory. 

For an AFM, the high temperature behavior is similar; 
thermal fluctuations destroy all correlations and E van- 
ishes. The low T behavior is different; the ground state 
of the AFM consists of alternating up and down spins. 
The spatial configurations thus store one bit of informa- 



tion about whether the odd or even sites are up. As can 



be seen in fig. 2, E ^ 1 as h^j 



0. 



For different couphngs and field strengths a range of 
E vs. hf^ relationships can be realized. E either shows a 
single maximum or decreases monotonically. It is always 
the case, though, that E is bounded from above by 1 — 
h^, which follows immediately if C^ is set equal to its 
maximum value, 1, in eq. (|^). 

Given that C^ was introduced as a measure of struc- 
ture, it is perhaps surprising that it behaves so differently 
from E. As /i^t increases, one might expect C^ to reach a 
maximum, as does E, and then decrease as the increas- 
ing thermalizing merges causal states that were distinct 
at lower temperatures. In fact, C^ increases monotoni- 
cally with h^. To understand this, recall that the causal 
states are the same for all T between zero and infinity. 
For the nn spin 1/2 Ising model, the number of causal 
states remains fixed at two. What does change as T is 
varied are the causal state probabilities. For the FM, 
as the temperature rises the distribution Pr(v5i) becomes 
more uniform, and C^ grows. This growth continues un- 
til T becomes infinite, since only there do the two causal 
states collapse into one, at which point C^t vanishes. 

For the AFM the situation is a little different. At T = 
there are two causal states corresponding to the two 
spatial phases of the alternating up-down pattern. The 
probability of these causal states is uniform; hence we 
see a low temperature statistical complexity of 1. At 
high (but finite) temperatures, the thermal fluctuations 
dominate; the anti-ferromagnetic order is lost, but the 
distribution over causal states is still relatively uniform 
so the statistical complexity remains large. (As with the 
FM, at T = cx) the two causal states merge and C^ jumps 
to zero.) Between these extremes there is a region where 
the influence of the external field dominates, biasing the 
configurations. This is reflected in a bias in the causal 
state probabilities and C^i dips below 1 as seen in fig. 2. 

The tendency for C^^ to remain large for large values 
of h^ is due to a more general effect, which follows from 
eq. (g): Cfj, = E + hfj,. The memory needed to model a 
process depends not only on the internal memory of the 
process, as measured by E, but also on its randomness, 
as measured by /i^. It is important to note, however, that 
C^ is driven up by thermalization not because the model 
attempts to account for random spins in the configura- 
tion. Rather, C^ rises with hfj, because Pr(iSi) becomes 
more uniform as the temperature increases. 

We have discussed three complementary statistics that 
as a whole capture the information processing capabilities 
embedded in spin systems. This framework has been ap- 
plied previously to the symbolic dynamics of continuous- 
state dynamical systems ||l| . The work presented here is 
the first exploration of thermal systems with these tools. 
In the dynamical systems studied, the statistical com- 
plexity varied as a function of h^ mainly due to changes 
in topological constraints on configurations. This led to 
changes in the number of causal states and in their con- 
nectivity. As a result, C^ has a unique maximum at [8] 



some /i^ < 1. (cf. Fig ??, ref. [y.) In sharp contrast, 
the thermal systems examined here have the same num- 
ber of causal states for all temperatures except zero and 
infinity. For all T 7^ thermal fluctuations are present: 
all configurations are possible and the connectivity of the 
causal states remains the same. This contrast points out 
a possibly useful distinction between deterministic and 
stochastic systems — a distinction that is lost by compar- 
ing these two different types of process solely in terms of 

These features and other work to be reported indicate 
that E and C^ capture properties that are different from 
existing thermodynamic quantities. Comparing the PM, 
FM, and AFM in terms of specific heat, for example, 
doesn't reveal the distinctions seen in fig. 2. This issue — 
along with analyses of 2D Ising systems, spin glasses, 
and recurrent neural networks — will be discussed else- 
where. For these higher dimensional systems, there are 
a number of ways to define E and C^. One approach 
is to consider infinite strips of spins as a single, infinite- 
dimensional spin. This method involves a natural ex- 
tension of the techniques developed here, yet we feel this 
might not faithfully capture the higher dimensional struc- 
ture present. Another approach is to adapt the cellular 
automata-theoretic formalism presented in ref. |R|. We 
shall examine both of these approaches in a future work. 
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